Software development usually requires testers to evaluate the quality of a software product in many ways including its requirements, specifications, user interface, and design. In the case of a platform product that is used by other programmers for creating software products, application program interfaces (APIs) also need to be evaluated as thoroughly as possible. One of the challenges in testing APIs is to perform a variety of tests to evaluate the quality of the product before shipping it. For example, one fundamental test is to verify that the parameters which may be passed in the invocation of an API are of the appropriate type and value. Another test may be a high-level test to verify that the invoked API behaves as intended. To perform this verification, the API may be included in an application such as a mail form, line of business application or the like so that its functionality may be evaluated.
It is not unusual for testers with a limited amount of time to attempt to use high-level testing, such as calling an API to perform a specific task, as a substitute for some fundamental testing like parameter testing. Although some level of parameter testing occurs during such scenario-type testing, it is difficult to quantify the level of coverage achieved by such testing. Each and every API should ideally be passed a varied set of values as parameters in order to achieve broad coverage in performing parameter validation testing. Historically, it has been difficult to achieve such coverage beyond the norm of about 60% of APIs tested for large platform products.
Testers are additionally challenged by the very nature of building a platform product. The platform product changes constantly as features and functionality are added and modified, sometimes in response to usability testing and customer feedback. Tests consequently break on a rather frequent basis whenever new APIs are introduced to the platform or existing APIs are changed or removed.
What is needed is a test environment that is capable of tracking code changes made during the development of a software product as the changes occur so that testers may better understand what tests may need to be changed for evaluating the quality of the current state of the product. Furthermore, this test environment should provide the capability for broad coverage in testing APIs and a way to quantify the level of such testing achieved for APIs.